The present invention relates generally to the field of haptics. More specifically, the present invention relates to methods and apparatuses for haptic interaction with video and image data.
Haptics is the field of touch sensation. One rapidly evolving area of computer technology is the development of haptic interfaces, that is, interfaces which provide a touch-sensitive link between a physical haptic device and an electronic environment. With a haptic interface, a user can obtain touch sensations of surface texture and rigidity of electronically generated virtual objects, such as may be created by a computer-aided design (CAD) system. Alternatively, the user may be able to sense forces as well as experience force feedback from haptic interaction with an electronically generated environment.
A haptic interface system typically includes a combination of computer software and hardware. The software component is capable of computing reaction forces as a result of forces applied by a user xe2x80x9ctouchingxe2x80x9d an electronic object. The hardware component is a haptic device that delivers and receives applied and reaction forces, respectively. Existing haptic devices include, for example, joysticks (such as are available from Immersion Human Interface Corporation, San Jose, Calif.; further information is available at www.immerse.com, the disclosure of which is incorporated herein by reference for all purposes), one-point probes (such as a stylus or xe2x80x9cspacepenxe2x80x9d) (such as the PHANToM(trademark) product available from SensAble Technologies, Inc., Cambridge, Mass.; further information is available at www.sensable.com, the disclosure of which is incorporated herein by reference for all purposes) and haptic gloves equipped with electronic sensors and actuators (such as the CyberTouch product available from Virtual Technologies, Inc., Palo Alto, Calif.; further information available at www.virtex.com, incorporated herein by reference for all purposes). The types of haptic interfaces available continue to be expanded, and their level of sophistication improved.
Haptic interfaces and devices have found application in a variety of research fields, as well as in some commercial products incorporating devices such as those noted above. Research has focussed on manipulating surface or volumetric models, or holographic objects. Commercial fields of application include medicine (e.g., simulation of laparoscopic surgery), product design (e.g., automotive styling), and entertainment (e.g., virtual reality, electronic games).
In all of the foregoing cases, the electronic objects or environments with which haptic interaction takes place are computer-generated. One of the most widely-used formats of entertainment and information is video, either through video tape or television. The present inventors have recognized that a viewer""s experience of the video medium could be enhanced by providing haptic interaction with objects and environment in a video image. However, as noted above, haptic interfaces have not yet been applied to video and image data. Haptic interaction has so far been limited to virtual objects or environments. Accordingly, the development of a system for haptic interaction with video and image data would be desirable.
The present invention meets this need by providing methods and systems for haptic interaction with video and image data. The methods and systems of the present invention receive video or image data, for example via television broadcast, cable or satellite television signal, or directly from a video player, and process the data to determine the geometry and material properties of various objects represented in the video image. Geometry and material property information may be deciphered from a video image by reference to various fixed data sources and/or the application of known methods for elucidating such information from video or image data, such as shape-from-motion or shape-from-shading techniques. Alternatively, all or some of this information may be provided to the system together with the video or image data via a preconfigured side channel. The system is able to coordinate and assemble the image data with the data relating to geometry and material properties of the video objects and provide a realistic haptic interaction with those objects to the user.
In one aspect, the present invention provides a method of haptically interacting with a video image. The method involves receiving a video image, determining geometry of one or more objects represented in the video image, and using a haptic display to interact with objects represented in the video image. The method may also include determining material properties of one or more objects represented in the video image, prior to the haptic interaction.
In another aspect , the invention provides a method of haptically interacting with a video image which includes receiving a video image, determining three-dimensional geometry of one or more objects represented in the video image, determining material properties of objects represented in the video image, elucidating a three-dimensional model for objects represented in the video image, and haptically interacting with the model.
In still another aspect, the invention provides a system for haptically interacting with a video image. The system includes a video display module, an image understanding module, a model recognizer module, and a haptic display module. The system may also include a video data signal decoder.